Sumo Logic Syslog Processor #1313

sumo-drosiek · 2021-04-22T10:51:25Z

Add Syslog Processor to keep backward compatibility with current solutions

Sumo Logic Syslog Processor

Supported pipeline types: logs

The Sumo Logic Syslog processor can be used to create attribute with facility name
basing on facility code. Default facility name is syslog.

Configuration

Field	Default	Description
facility_attr	facility	The attribute name in which a facility name is going to be written

Examples

Following table shows example facility names which are extracted from log line

log	facility
<13> Example log	user-level messages
<334> Another example log	syslog
Plain text log	syslog

Configuration Example

processors:
  sumologic_syslog:
    facility_attr: testAttrName

pmm-sumo

This is nice processor, one thing I am wondering if perhaps a slightly different approach could have been taken and "transformprocessor" could be created. It would be more generic and would allow to transform attributes

processor/sumologicsyslogprocessor/testdata/sumologic_syslog_config.yaml

pmalek-sumo · 2021-04-22T11:31:55Z

processor/sumologicsyslogprocessor/processor.go

+	}
+}
+
+func (ssp *sumologicSyslogProcessor) ProcessLogs(ctx context.Context, ld pdata.Logs) (pdata.Logs, error) {


Comment 🙏 ?

pmalek-sumo · 2021-04-22T11:32:38Z

processor/sumologicsyslogprocessor/processor.go

+				match := r.FindStringSubmatch(log.Body().StringVal())
+
+				if match != nil {
+					facility, _ := strconv.Atoi(match[1])


Why wouldn't we check the error 🤷 ?
https://golang.org/pkg/strconv/#Atoi

being a number is validated by regex

sumo-drosiek · 2021-04-22T11:38:51Z

@pmm-sumo We can iterate in that direction, but I don't have a general idea how this should work

pmm-sumo · 2021-04-22T11:42:31Z

@pmm-sumo We can iterate in that direction, but I don't have a general idea how this should work

By moving the mapping to config keys. E.g.

 attrtibute: facility_id
 mapped_attribute: facility_name
 mapping:
   0: "kernel_messages"
   1: "user-level messages"
   ...

sumo-drosiek · 2021-04-22T12:03:34Z

This case is not so simple, because I need to extract value for mapping, divide it by 8 and this is my base for mapping.
Anyway, mapping processor is a good idea and it's possible that we will need it anyway 😅 👍

pmm-sumo · 2021-04-22T12:14:07Z

I need to extract value for mapping, divide it by 8 and this is my base for mapping.

On the other hand, the mapping keys could be multiplied by 8, I guess? :)

sumo-drosiek · 2021-04-22T12:16:34Z

On the other hand, the mapping keys could be multiplied by 8, I guess? :)

It could, but it would look like pretty ugly 😰

 mapping:
   0: "kernel_messages"
   1: "kernel_messages"
   2: "kernel_messages"
   3: "kernel_messages"
   4: "kernel_messages"
   5: "kernel_messages"
   6: "kernel_messages"
   7: "kernel_messages"

I need rather ranges than exact values

github-actions · 2021-04-30T05:22:39Z

This PR was marked stale due to lack of activity. It will be closed in 7 days.

processor/sumologicsyslogprocessor/README.md

processor/sumologicsyslogprocessor/config.go

processor/sumologicsyslogprocessor/config_test.go

processor/sumologicsyslogprocessor/config.go

pmalek-sumo

🚀

github-actions · 2021-05-31T05:53:51Z

Closed as inactive. Feel free to reopen if this PR is still being worked on.

Signed-off-by: Dominik Rosiek <[email protected]> Co-authored-by: Patryk Małek <[email protected]>

Co-authored-by: Patryk Małek <[email protected]>

* add sumologicsyslogprocessor Signed-off-by: Dominik Rosiek <[email protected]> Co-authored-by: Patryk Małek <[email protected]>

Merge Sumo tail-sampling extensions Update source processor to new API Fix imports order Update deps for OTC v0.14.0 Expose tailsamplingprocessor metrics Config update (#289) CircleCI configuration update and linter fixes Tail sampling processor - update to v0.16 Source processor - update to v0.16 Tailsampling processor order fix Introduce cascading filter processor (#359) Introduce cascading filter processor Switch to AWS ECR (#409) Switch to AWS ECR Investigate issue with image tagging (#410) Move envs into $BASH_ENV AWS CLI orb Update README.md Clarify which span attributes are updated/added and `probablistic_filtering_ratio` behavior Provide invert (like grep -v) match capability (#509) Bump OTC-Sumo version Add Sumo examples (#681) Update README.md Update base dependency to v0.19.0-sumo Remove uneccesary return values per linter Switch master -> main Address linter errors in sourceprocessor Use cloud.namespace on AWS resourcedetection Update ec2.go TRACING-1684 | Change cloud.namespace tag from "ec2" to "aws/ec2". TRACING-1684 | Test fix Switch examples to use OTLP/HTTP (#864) Bump SumoLogic OTC base version Fix cascadingfilterprocessor unit test Introduce telegrafreceiver (#935) Increase testbed memory limits AWS OTel Collector templates (#788) Add AWS OTel Collector config templates Update core version to v0.22.0 AWS Distro for Opentelemetry collector configuration file (#983) Add AWS Distro Collector config Change endpoint, add insecure flag Update readme Update Telegraf for changes in core v0.22.0 Rebase on v0.24.0 upstream changes Add opentelemetry-collector-builder with Makefile and Github Action to build on PR (#1300) Add telegrafreceiver to opentelemetry-collector-builder config Add tracing-tests into pipeline (#1302) Install fluentbit (#1312) Add publish-check for all tags and branches Fix CI to actually produce artifacts for tags Update config.yml (#1412) Update aws-otel-config-file.yaml (#1537) Add resourcedetection Add additional receivers Sumo Logic Syslog Processor (#1313) add sumologicsyslogprocessor Signed-off-by: Dominik Rosiek <[email protected]> Co-authored-by: Patryk Małek <[email protected]> Update k8sprocessor with recent changes from sumologic-otel-collector

Merge Sumo tail-sampling extensions Update source processor to new API Fix imports order Update deps for OTC v0.14.0 Expose tailsamplingprocessor metrics Config update (#289) CircleCI configuration update and linter fixes Tail sampling processor - update to v0.16 Source processor - update to v0.16 Tailsampling processor order fix Introduce cascading filter processor (#359) Introduce cascading filter processor Switch to AWS ECR (#409) Switch to AWS ECR Investigate issue with image tagging (#410) Move envs into $BASH_ENV AWS CLI orb Update README.md Clarify which span attributes are updated/added and `probablistic_filtering_ratio` behavior Provide invert (like grep -v) match capability (#509) Bump OTC-Sumo version Add Sumo examples (#681) Update README.md Update base dependency to v0.19.0-sumo Remove uneccesary return values per linter Switch master -> main Address linter errors in sourceprocessor Use cloud.namespace on AWS resourcedetection Update ec2.go TRACING-1684 | Change cloud.namespace tag from "ec2" to "aws/ec2". TRACING-1684 | Test fix Switch examples to use OTLP/HTTP (#864) Bump SumoLogic OTC base version Fix cascadingfilterprocessor unit test Introduce telegrafreceiver (#935) Increase testbed memory limits AWS OTel Collector templates (#788) Add AWS OTel Collector config templates Update core version to v0.22.0 AWS Distro for Opentelemetry collector configuration file (#983) Add AWS Distro Collector config Change endpoint, add insecure flag Update readme Update Telegraf for changes in core v0.22.0 Rebase on v0.24.0 upstream changes Add opentelemetry-collector-builder with Makefile and Github Action to build on PR (#1300) Add telegrafreceiver to opentelemetry-collector-builder config Add tracing-tests into pipeline (#1302) Install fluentbit (#1312) Add publish-check for all tags and branches Fix CI to actually produce artifacts for tags Update config.yml (#1412) Update aws-otel-config-file.yaml (#1537) Add resourcedetection Add additional receivers Sumo Logic Syslog Processor (#1313) add sumologicsyslogprocessor Signed-off-by: Dominik Rosiek <[email protected]> Co-authored-by: Patryk Małek <[email protected]> Update k8sprocessor with recent changes from sumologic-otel-collector Add drop traces capability Fix linter issues Fix configschema go mods Update CircleCI Golang conf Remove old circleci config Remove windows-test target Update modules Update observability test from upstream Linter fix Update testbed limits Update number of processors Fix linter at sumologicsyslogprocessor Update infrastructure_service->platform Fix k8s linter check Increase testbed limits Fix double filling attributes Skip coverage Increase testbed limits Update Dockerfile Update testbed limits Update example, include better metrics and handle no-policies case Revert to old sourceprocessor config format Include missed testcase Fix linter lint & gofmt fix Use simpler rules in cascadingfilter Remove OnLateArrivingSpans Linter fix Increase memory ballasts Fix probability calculation Identation fix

sumo-drosiek requested review from pmm-sumo and pmalek-sumo April 22, 2021 10:51

pmalek-sumo force-pushed the main branch from 66cfcca to af1028b Compare April 22, 2021 11:13

pmm-sumo approved these changes Apr 22, 2021

View reviewed changes

pmalek-sumo reviewed Apr 22, 2021

View reviewed changes

pmalek-sumo force-pushed the main branch from af1028b to 43b7c05 Compare April 22, 2021 11:35

sumo-drosiek force-pushed the drosiek-sumo-syslog branch 3 times, most recently from a17e4fe to 1095887 Compare April 22, 2021 11:56

pmalek-sumo force-pushed the main branch from 43b7c05 to 5cc8712 Compare April 22, 2021 13:42

github-actions bot added the Stale label Apr 30, 2021

sumo-drosiek force-pushed the drosiek-sumo-syslog branch from 1095887 to d5978bb Compare May 4, 2021 06:36

sumo-drosiek requested a review from pmalek-sumo May 4, 2021 06:37

pmalek-sumo reviewed May 4, 2021

View reviewed changes

sumo-drosiek commented May 4, 2021

View reviewed changes

processor/sumologicsyslogprocessor/config.go Outdated Show resolved Hide resolved

pmalek-sumo approved these changes May 4, 2021

View reviewed changes

github-actions bot closed this May 31, 2021

sumo-drosiek reopened this May 31, 2021

Dominik Rosiek and others added 5 commits May 31, 2021 08:41

add sumologicsyslogprocessor

c20d623

Signed-off-by: Dominik Rosiek <[email protected]> Co-authored-by: Patryk Małek <[email protected]>

Update processor/sumologicsyslogprocessor/README.md

7cc6291

Co-authored-by: Patryk Małek <[email protected]>

Update processor/sumologicsyslogprocessor/config_test.go

22417a5

Co-authored-by: Patryk Małek <[email protected]>

Update processor/sumologicsyslogprocessor/config_test.go

62ea98c

Co-authored-by: Patryk Małek <[email protected]>

Update processor/sumologicsyslogprocessor/config.go

a26c72f

Co-authored-by: Patryk Małek <[email protected]>

Update processor/sumologicsyslogprocessor/config.go

15fea9b

sumo-drosiek force-pushed the drosiek-sumo-syslog branch from 2280e65 to 15fea9b Compare May 31, 2021 06:41

sumo-drosiek merged commit e095546 into SumoLogic:main May 31, 2021

sumo-drosiek deleted the drosiek-sumo-syslog branch May 31, 2021 06:44

sumo-drosiek added a commit that referenced this pull request May 31, 2021

Sumo Logic Syslog Processor (#1313)

7f8608d

* add sumologicsyslogprocessor Signed-off-by: Dominik Rosiek <[email protected]> Co-authored-by: Patryk Małek <[email protected]>

sumo-drosiek added a commit that referenced this pull request May 31, 2021

Sumo Logic Syslog Processor (#1313)

c20d589

* add sumologicsyslogprocessor Signed-off-by: Dominik Rosiek <[email protected]> Co-authored-by: Patryk Małek <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sumo Logic Syslog Processor #1313

Sumo Logic Syslog Processor #1313

sumo-drosiek commented Apr 22, 2021

pmm-sumo left a comment

pmalek-sumo Apr 22, 2021

pmalek-sumo Apr 22, 2021

sumo-drosiek Apr 22, 2021 •

edited

Loading

sumo-drosiek commented Apr 22, 2021

pmm-sumo commented Apr 22, 2021

sumo-drosiek commented Apr 22, 2021

pmm-sumo commented Apr 22, 2021

sumo-drosiek commented Apr 22, 2021 •

edited

Loading

github-actions bot commented Apr 30, 2021

pmalek-sumo left a comment

github-actions bot commented May 31, 2021

Sumo Logic Syslog Processor #1313

Sumo Logic Syslog Processor #1313

Conversation

sumo-drosiek commented Apr 22, 2021

Sumo Logic Syslog Processor

Configuration

Examples

Configuration Example

pmm-sumo left a comment

Choose a reason for hiding this comment

pmalek-sumo Apr 22, 2021

Choose a reason for hiding this comment

pmalek-sumo Apr 22, 2021

Choose a reason for hiding this comment

sumo-drosiek Apr 22, 2021 • edited Loading

Choose a reason for hiding this comment

sumo-drosiek commented Apr 22, 2021

pmm-sumo commented Apr 22, 2021

sumo-drosiek commented Apr 22, 2021

pmm-sumo commented Apr 22, 2021

sumo-drosiek commented Apr 22, 2021 • edited Loading

github-actions bot commented Apr 30, 2021

pmalek-sumo left a comment

Choose a reason for hiding this comment

github-actions bot commented May 31, 2021

sumo-drosiek Apr 22, 2021 •

edited

Loading

sumo-drosiek commented Apr 22, 2021 •

edited

Loading